separateDebugInfo: add symlinks to executable and source for debuginfod support#394674
Conversation
|
Currently this only allows serving unpatched source. If source files are patched, debuginfod servers can only serve the unpatched source.
|
|
The PR's base branch is set to staging, but 89 commits from the master branch are included. Make sure you know the right base branch for your changes, then:
|
59845de to
dfbb727
Compare
77f7676 to
28cf81f
Compare
|
I added logic to copy patched files to the debug output so that the debuginfod server can serve them. the symlink to the source creates new references, and copying patched files might create more. This is thus incompatible with allowedReferences et at. I added some logic that disables these attributes but for the debug output only. this required structured attrs. With grep I determined that only a dozen derivations are affected in nixpkgs. I also encountered issues with autopatchelfhook, see the commit message for the exact issue. I taught autopatchelfhook to ignore the debug output. |
|
Re-running CI due to a odd failure fixed in #416448 |
mweinelt
left a comment
There was a problem hiding this comment.
Looks and sounds plausible. I think we should give it a try.
Please fix the scoping of the commit message for the commit that adds the release notes entry.
…od support
A debuginfod support must be able to map a build-id to
- debug symbols
- the original elf file for which the debug symbols where separated
- the corresponding source files
Currently, hydra provides an index from build-id to the nar of the debug
output containing the debug symbols.
Add symlinks in these outputs so that we can recover the store path of
the source and original elf file. We can then fetch them by the normal
binary cache protocol.
About source files: to minimize storage demands, in the ideal case,
software would be built from the source store path $src and the
debuginfod server would just have to serve source files from this store
path. In practice, source files are sometimes patched as part of the
build. This commit stores the modified files in the debug output is a so
called source overlay so that the debuginfod serve can serve the patched
content of the file.
The checksum was chosen as follows (where big is 4GB of zeros):
$ hyperfine -L s sysv,bsd,crc,sha1,sha224,sha256,sha384,sha512,blake2b,sm3 'cksum -a {s} big'
Benchmark 1: cksum -a sysv big
Time (mean ± σ): 854.5 ms ± 270.5 ms [User: 245.3 ms, System: 601.8 ms]
Range (min … max): 760.5 ms … 1623.8 ms 10 runs
Warning: The first benchmarking run for this command was significantly slower than the rest (1.624 s). This could be caused by (filesystem) caches that were not filled until after the first run. You should consider using the '--warmup' option to fill those caches before the actual benchmark. Alternatively, use the '--prepare' option to clear the caches before each timing run.
Benchmark 2: cksum -a bsd big
Time (mean ± σ): 5.838 s ± 0.045 s [User: 5.118 s, System: 0.693 s]
Range (min … max): 5.767 s … 5.897 s 10 runs
Benchmark 3: cksum -a crc big
Time (mean ± σ): 829.9 ms ± 28.6 ms [User: 274.5 ms, System: 551.0 ms]
Range (min … max): 803.2 ms … 904.8 ms 10 runs
Benchmark 4: cksum -a sha1 big
Time (mean ± σ): 2.553 s ± 0.010 s [User: 1.912 s, System: 0.631 s]
Range (min … max): 2.543 s … 2.575 s 10 runs
Benchmark 5: cksum -a sha224 big
Time (mean ± σ): 2.716 s ± 0.018 s [User: 2.054 s, System: 0.645 s]
Range (min … max): 2.695 s … 2.743 s 10 runs
Benchmark 6: cksum -a sha256 big
Time (mean ± σ): 2.751 s ± 0.029 s [User: 2.057 s, System: 0.674 s]
Range (min … max): 2.712 s … 2.812 s 10 runs
Benchmark 7: cksum -a sha384 big
Time (mean ± σ): 5.600 s ± 0.049 s [User: 4.820 s, System: 0.753 s]
Range (min … max): 5.515 s … 5.683 s 10 runs
Benchmark 8: cksum -a sha512 big
Time (mean ± σ): 5.543 s ± 0.021 s [User: 4.751 s, System: 0.768 s]
Range (min … max): 5.523 s … 5.579 s 10 runs
Benchmark 9: cksum -a blake2b big
Time (mean ± σ): 5.091 s ± 0.025 s [User: 4.306 s, System: 0.764 s]
Range (min … max): 5.048 s … 5.125 s 10 runs
Benchmark 10: cksum -a sm3 big
Time (mean ± σ): 14.220 s ± 0.120 s [User: 13.376 s, System: 0.783 s]
Range (min … max): 14.077 s … 14.497 s 10 runs
Summary
cksum -a crc big ran
1.03 ± 0.33 times faster than cksum -a sysv big
3.08 ± 0.11 times faster than cksum -a sha1 big
3.27 ± 0.11 times faster than cksum -a sha224 big
3.31 ± 0.12 times faster than cksum -a sha256 big
6.13 ± 0.21 times faster than cksum -a blake2b big
6.68 ± 0.23 times faster than cksum -a sha512 big
6.75 ± 0.24 times faster than cksum -a sha384 big
7.03 ± 0.25 times faster than cksum -a bsd big
17.13 ± 0.61 times faster than cksum -a sm3 big
unfortunately, crc (and sysv) are not supported by --check, so they are
disqualified. sha1 sha224 and sha256 are sensibly as fast as one
another, so let's use a non broken one, even though cryptographic
qualities are not needed here.
needed for separateDebugInfo
needed for separateDebugInfo = true
needed by separateDebugInfo = true
neeeded by separateDebugInfo = true
needed by separateDebugInfo = true
… by separateDebugInfo this output contains unpredictable references by design. It would nearly always break allowedReferences et al. As it's not intended to enter a real closure, but to be fetched on demand by buildid, disable reference whitelisting instead, but only for this output. This requires __structuredAttrs. Only a dozen derivations are affected in the whole of nixpkgs.
required for separateDebugInfo
otherwise systemd fails to build because autoPatchelfHook patches libs
with debug objects and this creates circular dependencies from out to
debug (of course, debug depends on out)
example of faulty behavior:
searching for dependencies of /nix/store/c8cd7b82py02f0rkags351nhg82wwjm6-systemd-minimal-257.5/bin/systemd-delta
libsystemd-shared-257.so -> found: /nix/store/icpqrawjhsw4fbi4i2hp7cxvf3kbzq7m-systemd-minimal-257.5-debug/lib/debug
libc.so.6 -> found: /nix/store/7lyblga0bzjk0sl93vp4aiwbvb57vp2x-glibc-2.40-66/lib
|
Did this break jetbrain jdk? EDIT: yep, setting fixes jetbrains.jdk build for me |
|
I'm currently building this, but this should fix it: #425529 |
|
FYI this caused some regressions in Lix that are quite hard for us to fix because Now, we could just backport it and roll some releases, but that's kind of annoying, so I'm just going to point out that this has some annoying out-of-tree effects here in case there's a desire to fix it another way :) |
|
This pull request has been mentioned on NixOS Discourse. There might be relevant details there: https://discourse.nixos.org/t/unstable-fails-to-build-stable-ok-flakes/67837/2 |
For debuginfod support we must be able to map a build-id to
Currently, hydra provides an index from build-id to the nar of the debug output containing the debug symbols.
Add symlinks in these outputs so that we can recover the store path of the source and original elf file. We can then fetch them by the normal binary cache protocol.
Pros:
Cons
A proof of concept of such debuginfo server is available at https://github.com/symphorien/nixseparatedebuginfod2
Things done
nix.conf? (See Nix manual)sandbox = relaxedsandbox = truenix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage./result/bin/)Add a 👍 reaction to pull requests you find important.